Review of surgical robotic systems for keyhole and endoscopic procedures: state of the art and perspectives

Abstract

Minimally invasive surgery, including laparoscopic and thoracoscopic procedures, benefits patients in terms of improved postoperative outcomes and short recovery time. The challenges in hand–eye coordination and manipulation dexterity during the aforementioned procedures have inspired an enormous wave of developments on surgical robotic systems to assist keyhole and endoscopic procedures in the past decades. This paper presents a systematic review of the state-of-the-art systems, picturing a detailed landscape of the system configurations, actuation schemes, and control approaches of the existing surgical robotic systems for keyhole and endoscopic procedures. The development challenges and future perspectives are discussed in depth to point out the need for new enabling technologies and inspire future researches.

Keywords

surgical robots / keyhole surgery / endoscopic surgery

Introduction

Keyhole surgery (e.g., laparoscopic and thoracoscopic procedures) has been widely adopted due to its advantages, such as low postoperative complication rates, less pain, short recovery time, and excellent cosmesis [¹,²]. Manipulation challenges in these procedures, including limited visual perception, reduced distal dexterity, reversed hand–eye coordination, and hindered haptic sensing, have inspired a vast wave of developments on surgical robotic systems to improve their assistance and obtain superhuman capabilities [³–⁵].

Robotic systems for keyhole and endoscopic procedures usually consist of a patient-side cart and a surgeon console (Fig. 1). As presented in the “Configuration and actuation of the patient-side cart” section, the patient-side cart often involves a few surgical manipulators that maneuver a laparoscope (or a thoracoscope) and two to three surgical instruments. A laparoscope and thoracoscope are the same in their composition. During a procedure, a laparoscope is inserted into the abdomen, whereas a thoracoscope is inserted into the thorax. Surgical manipulators can have various forms and actuation schemes. Given that the control scheme of surgical robotic systems is mainly teleoperation, the surgeon console primarily consists of a 2D/3D display, a pair of master devices, and a touchscreen/keyboard with a few pedals for user inputs. The system sometimes includes a device cart for additional display, electrical surgical equipment, and data processing computers serving as an information hub during surgery.

Surgeries other than keyhole and endoscopic procedures utilize different treatment techniques, which in turn lead to surgical robotic systems with different characteristics. For example, orthopedic and neurosurgical robots emphasize on accurate registration and constrained intraoperative path planning [⁶,⁷], whereas robotic percutaneous interventional procedures focus on instrument compactness (e.g., via the use of concentric tubes [⁸,⁹]) and magnetic resonance imaging compatibility [¹⁰,¹¹].

This paper presents a comprehensive review of the state-of-the-art surgical robotic systems for laparoscopic, thoracoscopic, and endoscopic procedures. The search methodology is reported in the “Search methodology” section. The system configurations, actuation schemes, and design considerations are analyzed in the “Configuration and actuation of the patient-side cart” section, and additional sensors in robotic surgery for haptic sensation and visual perception are discussed in the “Additional sensors in robotic surgery” section. The control approaches, including teleoperation, surgical automation, and system autonomy, are elaborated in the “Control approaches” section. Future developments and perspectives are discussed in the “Challenges and future perspectives” section to inspire future studies. The conclusion is provided in the “Conclusions” section.

Existing survey papers either span too broadly or only focus on a specific topic. For example, survey papers [³,⁵,¹²] cover a wide spectrum of advancements in surgical robotics, including orthopedics, neurosurgery, laparoscopy, catheterization, and percutaneous procedures. These papers fail to systematically introduce the state-of-the-art achievements of surgical robotic systems for keyhole and endoscopic procedures. On the other hand, review papers (e.g., References [⁶,⁷,¹¹,¹³,¹⁴]) either focus on particular applications (e.g., digestive procedures, arthroplasty, neurosurgery, and urology) or design approaches (e.g., via the use of continuum mechanisms). Surgical robotic systems for keyhole and endoscopic procedures share some common characteristics, such as dual-arm manipulation, visual guidance, and teleoperation. This review paper attempts to help readers form a systematic and comprehensive perception about the state-of-the-art achievements such that future studies can be planned beyond the achievements.

Search methodology

This literature review focuses on the enabling technologies and system integrations of surgical robots for keyhole and endoscopic procedures. The searches were performed on Thomson Reuters Web of Science Core Collection and IEEE Xplore to find relevant literatures in English. The searches in Thomson Reuters Web of Science Core Collection were specified in the areas of robotics and engineering using the advanced search option, whereas those in the IEEE database were conducted using the free-text protocol. The search terms applied are listed in Table 1 for each individual subsection as these subsections are relatively standalone. Literatures with at least five average annual citations within the first 100 most-cited papers were identified. Additional relevant records were included from the authors’ literature library.

The authors reviewed the abstracts of the identified literatures to exclude records in irrelevant topics (e.g., orthopedics, vascular intervention, or neurology), technologies for manual tool designs, surgical platform descriptions with no or little implementation details, clinical reports with no or little engineering details, and results with limited practical significance. Considering that this review does not intend to exhaustively include all relevant papers, sometimes only representative articles and milestone works (e.g., one or two most cited papers) were included. All the included literatures were then presented according to the structure of this paper. This selection process was based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria and followed the PRISMA flowchart in Fig. 2 for each subsection. The applied search terms and statistics for each subsection are summarized in Table 1. A total of 219 papers were included in the “Configuration and actuation of the patient-side cart,” “Additional sensors in robotic surgery,” and “Control approaches” sections in this review.

Configuration and actuation of the patient-side cart

A patient-side cart often consists of a few surgical manipulators. As shown in Fig. 3A, the surgical manipulators maneuver a laparoscope (or a thoracoscope) and two to three straight surgical instruments with distal wrists through a few incisions made on the abdominal or thoracic wall to visualize the surgical site and perform treatments. The distal wrists integrated in the surgical instruments are mainly used to increase the distal dexterity (i.e., the ability of orienting the surgical end effector as desired).

The paradigm in Fig. 3A involves a few skin incisions, and it is referred to as a multi-port procedure. In this procedure, the surgical manipulators shall realize remote-center-of-motion (RCM) movements or multiple degree-of-freedom (DoF) intracorporeal motions. These manipulators with different forms and actuation schemes are discussed in the “RCM movements” and “Intracorporeal movements” sections, respectively.

The desire of further reducing surgical invasiveness leads to the proposal of single-port procedures [¹⁵] and natural orifice transluminal endoscopic surgery (NOTES) [¹⁶]. In single-port procedures, surgical instruments have to be inserted into the abdomen through a single incision to access the surgical site. It is challenging for surgeons to become proficient in crossed hand–eye coordination; thus, robotic assistance is needed. The inserted instruments can be arranged in two ways: the X configuration as shown in Fig. 3B and the Y configuration as shown in Fig. 3C. In the X configuration, the technical approaches from the multi-port surgical system may be directly applied, but collisions between the surgical manipulators and the instruments should be carefully avoided as the instruments’ workspace is mutually limited. In the Y configuration, the surgical instruments shall be unfolded to form a working pose. Collisions are minimized, but the instruments’ payload capability is of concern because the external load has a much longer moment arm compared with the moment arm of the actuation force from the cable inside the instrument’s stem. The existing state-of-the-art systems for single-port procedures are summarized in the “Systems for single-port procedures” section.

NOTES procedures access a surgical site via confined and curved natural orifices and can potentially avoid the use of any skin incision. However, the difficulty in instrumentation limits the application of NOTES procedures. Representative endoscopic and NOTES systems are summarized in the “Systems for endoscopic and NOTES procedures” section.

Systems for multi-port procedures

Surgical instruments are individually inserted through the incisions in the abdominal or thoracic wall during a multi-port procedure. They can either have a straight stick-like form with a distal wrist or a multi-DoF structure.

In the former case, the instruments shall be manipulated by the patient-side manipulators for RCM movements, where the instrument is pivoted with respect to the skin incision in order to avoid tearing the abdominal wall. Then, the instrument is given four DoFs by the extracorporeal manipulator, including the pitch and yaw DoFs, as well as the translation and rotation along and about the instrument’s axis. The forms and structures of RCM manipulators are reviewed in the “RCM movements” section. A distal wrist is often integrated to enhance the distal dexterity (i.e., increase the number of DoFs). Various wrist designs are reported in the “Wrist design and actuation” section.

A number of designs on multi-DoF surgical instruments are reviewed in the “Intracorporeal movements” section. These instruments realize dexterous intracorporeal movements without using RCM manipulators.

RCM movements

In the multi-port procedure, a stick-like instrument requires RCM movements: it can be tilted in the pitch and yaw directions with respect to the skin incision. These RCM motions are realized by the patient-side surgical manipulators, either by using an RCM mechanism or via controlling a manipulator’s multiple joints in a coordinated manner (i.e., programmable RCM movements). Several RCM mechanisms are shown in Fig. 4.

In the parallelogram-based RCM mechanism in Fig. 4A with the representative implementations in the da Vinci system [¹⁷] and the REVO-I system [¹⁸], the rotation from the actuator at position A is transmitted to position O via the two parallelograms. The second actuator at position B enables a rotation about the BO axis. To reduce the mass of the RCM mechanisms so as to ease the dynamic control of the instrument’s movements, timing belts [¹⁹] and cables [²⁰,²¹] are used to realize equivalent parallelograms for RCM movements as shown in Fig. 4B. However, cable elongation under loads can be challenging for the manipulator accuracy. To increase the structural rigidity, a parallel structure is used as shown in Fig. 4C, where two prismatic actuators are used to provide the 2-DoF rotation about the RCM point [²²].

The serially connected spherical linkage as shown in Fig. 4D can also be used [²³,²⁴]. To further increase the stiffness, a parallel spherical linkage was proposed as shown in Fig. 4E [²⁵]. The design in Fig. 4E realizes 3-DoF motions, including a rotation about the instrument axis. This axial rotation is usually not realized by the RCM mechanisms, for example, the ones in Fig. 4A–4D and 4F.

RCM mechanisms based on goniometer arc tracks are also possible candidates [²⁶,²⁷]. However, the arc length is proportional to the angular motion range, and large motion ranges lead to bulky designs and inconvenience in patient-side deployment.

Programmable RCM movements are realized via synchronized control of a manipulator’s multiple joints. The manipulators can have either serial [²⁸–³¹], parallel [³²–³⁴], or hybrid [³⁵,³⁶] structures.

The control for programmable RCM movements can be realized using a geometrical approach [²⁸] or instantaneous kinematics. The latter mainly includes the use of (1) null space projection [³¹,³⁷,³⁸] or (2) extended Jacobian methods [³⁶,³⁹–⁴¹]. Dynamic control methods that minimize the contact force between the instrument and the trocar can also be used [⁴²,⁴³]. Compared with RCM mechanisms, manipulators with programmable RCM movements provide convenience during the preoperative docking of the instruments. However, RCM mechanisms are considered safer and more reliable due to the mechanically constrained movements.

Besides the two aforementioned RCM approaches, passive RCM technique can also be adopted [⁴⁴–⁴⁶], wherein the extracorporeal manipulator is underactuated and the incision point helps to determine the pose of the inserted instrument. Passive RCM technique is safer, but the instrument’s accuracy is often affected by the compliance from the incision port in the abdominal wall under the pneumoperitoneum.

Wrist design and actuation

While the RCM movements of an instrument are realized by an extracorporeal manipulator, a wrist is often integrated at the distal end for dexterity enhancement such that suturing and knot tying can be more conveniently conducted.

Many wrists adopt serial-structured designs. The desire for design compactness and proximal actuator arrangement often leads to the choice of cable actuation [¹⁷,²⁴,⁴⁷,⁴⁸], including the famous EndoWrist design shown in Fig. 5A. Shape memory alloy actuation is also a possible approach to realize compact wrist designs [⁴⁹,⁵⁰]. However, the motion responses are relatively slow. For example, the thermal exchange took approximately 8 s to complete in Reference [⁴⁹].

To enhance the wrist’s structural rigidity, serial and parallel linkages are also proposed. Representative examples using serially connected coupler actuation [⁵¹] and a 3-PRS structure [⁵²] are shown in Fig. 5B and Fig. 5C, respectively. However, these linkage-actuated wrists may have limited motion ranges. For example, motion ranges of the pitch and yaw joints are only±40° in Reference [⁵¹], whereas the parallel-linkage design in Reference [⁵²] has a pitch motion from −50° to 70° and a yaw motion of±64°. In comparison, the EndoWrist in the da Vinci system has a pitch motion range of±70° and a yaw motion range of±90° [¹⁷].

The pulleys used in the cable-driven wrists and the pinned joints in the linkage-driven wrists have limited potentials in further miniaturization. Continuum mechanisms, which are coined in Reference [⁵³] and transmit forces and motions via the structures’ continuous deformations, have been explored. Examples of continuum wrists for multi-port procedures using a 2-DoF bending segment are shown in Figs. 5D [²⁷] and 5E [⁵⁴], featuring a multi-backbone design and a concentric-tube design, respectively.

The structural and modeling simplicity makes the design popular in many surgical robotic systems for single-port and NOTES applications [⁵⁵–⁵⁷]. To further improve the structural simplicity, a deformable wrist design was recently proposed as shown in Fig. 5F [⁵⁸]. The rigidity is also enhanced as elastic strips were used to replace elastic wires. In these continuum wrists, fatigue may seem a problem due to the repetitive bending. However, superelastic nitinol can easily undergo 105 cycles of deformation [⁵⁹], which fulfill the requirement for a multi-use instrument.

Intracorporeal movements

Besides stick-like instruments with an RCM manipulator, the patient-side surgical manipulator can also be designed to directly realize multi-DoF intracorporeal movements. In this case, only a lockable stand is required to hold these dexterous surgical manipulators to the entry ports of a patient’s abdomen. Then, the manipulators will no longer undergo RCM movements (e.g., swinging back and forth), eliminating the risks of mutual collisions of these bedside manipulators.

Surgical manipulators with multiple intracorporeal DoFs can have either articulated, continuum, or hybrid structures, as shown in Fig. 6.

In articulated designs, cable actuation [⁴⁷] or embedded motors [²⁸,⁴⁸] can be used. However, the use of embedded motors often results in bulky designs, and cable actuation is affected by tension keeping. Thus, the continuum structure has recently become a popular choice, where cable tension keeping can be eased by the elastic structure [⁵⁴,⁶⁰], or avoided in multi-elastic-backbone designs [⁶¹,⁶²]. A hybrid design using a 2-DoF inverted dual continuum mechanism for intracorporeal translation and a 2-DoF EndoWrist for orientation has recently been proposed; this design combines the advantages of the articulated wrist that provides enhanced dexterity in confined spaces and the dual continuum mechanism that provides enhanced payload and reliability [⁶³].

Through the use of intracorporeal manipulators, the possibility of collision among the extracorporeal RCM manipulators is minimized. However, an intracorporeal manipulator should be fully inserted to deploy all its joints for multi-DoF movements. Its dexterity is hence limited at regions close to the abdominal wall. For wider clinical applications, most robotic systems for multi-port procedures still adopt the approach with RCM manipulators.

Systems for single-port procedures

General laparoscopic and thoracoscopic single-port procedures were introduced for reducing surgical invasiveness at the cost of increased manual manipulation difficulties and instrumentation complexity. Robotic assistance was introduced, and the adopted actuation schemes include cable/tendon actuation, embedded motorization, linkage-based transmission, and continuum mechanisms.

Cable actuation was adopted in the X and Y configurations, as shown in Fig. 3. The examples adopting the X configuration include the da Vinci Single-Site video endoscopic single-port access (VeSPA) surgical platform [⁶⁴] and the Samsung single-incision surgical system [⁶⁵]. To avoid manipulator collisions and instrument interference, many systems adopted the Y configuration, including the da Vinci SP system [⁵⁶], the Single Port Orifice Robotic Technology (SPORT) system [⁶⁶], the SurgiBot system [⁶⁷], and the single-port system with a flexible access tube [⁶⁸]. Some examples are shown in Fig. 7A. Cable/tendon actuation is a relatively mature technique. Ideally, pulleys should be applied at the manipulator’s joints to improve reliability and transmission smoothness. However, the pulley size becomes a major limit for further miniaturization of a multi-joint instrument. For this reason, some designs do not use pulleys and let the actuation cables slide against the rounded edges of the structural components. Besides cable wear, the introduced friction can cause actuation hysteresis and affect movement accuracy. In all the designs, the actuation cables shall be arranged distant enough from the joint axis to generate enough forces at the instrument’s distal end. Thus, most existing systems have an access tube diameter larger than 25 mm.

The following systems that use embedded miniature motors adopted the Y configuration: Single-Port lapaRoscopy bImaNual robot (SPRINT) system (Fig. 7B) [⁶⁹], Robot-Assisted Surgical Device (RASD) system (Virtual Incision Corporation, Lincoln, Nebraska, USA), the Single-Incision in vivo Surgical Robot (SISR) system [⁷⁰], and the NISI single-port robotic system [⁷¹]. The use of embedded motors inside the manipulator provides convenient modular joint designs. However, a sufficient motor power rating leads to a relatively large manipulator diameter. Most existing surgical systems adopting this approach need an access port larger than 30 mm in diameter. The sterilizability of the embedded motors may also increase the system costs.

Linkage can also be used for manipulator design in a single-port system. Representative examples include the single-port surgery (SPS) system (Fig. 7C) [⁷²] and PLAte-spring mechanism-based LAparoscopic Surgical robot (PLAS) system [⁷³]. The linkage-based manipulators generate relatively good payload performance. However, the inherent difficulties in designing multi-DoF spatial linkages, such as transmission and interference avoidance, often lead to limited instrument distal dexterity. For example, the maximal joint angle is only 45° in the SPS system [⁷²]. In addition, miniaturization of the manipulator may be limited by the size of the structural hinges. Most linkage-based systems require an incision diameter larger than 25 mm [⁷²,⁷³].

Continuum mechanisms transmit force and movement via structural deformation. Hence, all structural members undertake dual roles of structure and transmission. Using continuum mechanism to design a single-port surgical robot can therefore achieve better design compactness. Examples using this design include the Insertable Robotic Effector Platform (IREP) system with a 15 mm stem diameter [⁷⁴] and the SJTU Unfoldable Robotic System (SURS) with a 12 mm stem diameter (Fig. 7D) [⁵⁷]. In addition, the dual continuum mechanism proposed in Reference [⁵⁷] introduces actuation modularity and substantially increases the payload capability of the continuum surgical manipulator, making it an appealing candidate for future surgical robotic products. In the clinical scenario where only low payload capability is needed (e.g., performing ablation), a cardiothoracic endoscopic soft surgical robot was proposed [⁷⁵].

Systems for endoscopic and NOTES procedures

The delivery of surgical tools through narrow and curved natural orifices creates strict constraints on the tool’s distal dexterity and payload capability [⁷⁶]. The application of endoscopic and NOTES procedures is hence limited due to the challenges in tool instrumentation [⁷⁷]. Existing systems usually adopt the designs with articulated or continuum structures.

In articulated designs, structural compactness is of primary consideration. Besides cable actuation [⁷⁸,⁷⁹], surgical manipulators can also use embedded miniature motors [⁸⁰–⁸³]. The use of embedded miniature motors leads to beneficial design modularity and even system reconfigurability. These robots can move away from the cavity entrance and are typically magnetically anchored on the abdominal wall from an outside dock. However, further miniaturization of such robots is challenging due to various integrated actuation components.

Continuum robots, on the other hand, incorporate elastic structures and tendon/backbone actuation schemes, leading to a compact and flexible design [⁵⁵,⁸⁴–⁹⁰]. Two representative examples are shown in Fig. 8C. However, due to the inherent compliance of continuum structures, the payload capability is often deteriorated with an increased total length of the arm.

Additional sensors in robotic surgery

To further enhance the safety and functionality of a surgical robotic system, additional sensors were integrated. Two major categories are primarily involved: force sensing and supplementary visual modalities.

Force sensing

Force sensing in surgical robotic systems can generate faithful force feedback to the operating surgeon and can potentially help increase the safety of typical surgical tasks, including tissue manipulation [⁹¹] and blunt dissection [⁹²].

Force sensing during surgery mainly include two types: (1) measuring the tissue gripping force and (2) measuring the interaction force with environment. Sensing approaches include extrinsic force sensing and intrinsic force sensing. Extrinsic force sensing integrates a sensing element at the location of contact and directly measures the force, whereas intrinsic force sensing, proposed in Reference [⁹³], uses actuator-level information to calculate the contact force, as shown in Fig. 9C. However, the former may provide a more accurate result. The increased complexity of sensor-integrated instruments may lead to reduced reliability, providing challenges to sterilization and higher costs.

Gripping force sensing using the extrinsic approach includes the use of a strain gauge [⁹⁴], polyvinylidene difluoride (PVDF) elements [⁹⁵], fiber Bragg grating (FBG) [⁹⁶], and capacitive sensing cells (Fig. 9A) [⁹⁷]. Given that a gripping force is always actively applied, the intrinsic sensing approach might be more suitable. Representative examples include measuring the driving shaft tensions [⁵¹], driving pulley strains [⁹⁸], driving pulley torques [⁹⁹], and motor currents [¹⁰⁰].

Interaction force sensing with environment adopting the extrinsic approach includes the use of strain gauges [¹⁰¹,¹⁰²] with one design (Fig. 9B) [¹⁰¹], optical intensity [¹⁰³,¹⁰⁴], FBG [¹⁰⁵], and PVDF piezoelectric film [¹⁰⁶]. On the other hand, intrinsic sensing is implemented by measuring the backbone actuation forces [¹⁰⁷], motor currents [¹⁰⁸], and actuation pulley strains [⁹⁸]. Compared with the extrinsic sensing approach, the intrinsic sensing approach allows versatile extracorporeal sensor arrangement, which enables structural simplification of the inserted instruments. However, intrinsic force sensing through actuation cables or motor currents is affected by friction and moving inertia. Thus, the accuracy may be limited.

Supplementary visual modality

Endoscopic imaging is the basic visual modality in keyhole surgery, and it provides a direct view of the targeted surgical site. Besides endoscopic imaging, other visual modalities are often integrated to provide richer intracorporeal information at the anatomical, tissue, or cellular levels to improve visualization and facilitate surgical treatments.

Surface reconstruction

Surface reconstruction provides spatial information about the tissue surface geometry, which can be used for depth perception and surgical navigation [¹¹⁰,¹¹¹]. Several approaches can be used for 3D surface reconstruction of a surgical scene intraoperatively, including the use of stereoscopes [¹¹²–¹¹⁴] or structured lights [¹¹⁵–¹¹⁷]. However, the miniaturization can be challenging due to a minimal requirement of a baseline distance [¹¹⁸].

The 3D shape of the environment can also be obtained from the motion of a single endoscope, including the Shape-from-Motion (SfM) [¹¹⁹] and Shape-from-Shading (SfS) techniques [¹²⁰]. In SfM, a 3D model template is first constructed from multiple views of the tissue, and tissue deformation is detected from the motions of the point cloud. In SfS, the depth of each pixel is computed by relating the pixel’s brightness to its normal surface direction with a reflectance model. These techniques can be readily applied with current laparoscopic hardware, but the performances rely on the feature correspondence or the validity of the reflectance model [¹¹⁸].

Simultaneous localization and mapping (SLAM), which updates an environment map via a camera’s view while tracking the camera’s location, has also been utilized in surgical surface reconstruction [¹²¹,¹²²]. SLAM is appealing in the surgical scene due to its real-time capability even while using a standard laparoscope. Recent works that addressed the challenges such as tracking deformable tissues and robust feature matching [¹²³,¹²⁴] have made SLAM even more promising (Fig. 10A).

Fluorescence and spectral imaging

Near-infrared fluorescence (NIRF) imaging improves contrasted views of specific surgical sites against surrounding tissues. Near-infrared light (700–900 nm) can travel up to centimeters through tissues to reach the fluorescent contrast agents, which have been developed for different types of targets according to the absorption and scattering properties of the tissue components [¹²⁸]. The applications of NIRF include sentinel lymph-node mapping, identification of vascular and biliary anatomy, and assessment of organ and tissue perfusion [¹²⁵] (Fig. 10B1). Fluorescence is integrated into the da Vinci surgical system and was found to be highly useful [¹²⁹]. Specific surgical procedures employing NIRF include partial nephrectomy [¹³⁰], cholecystectomy [¹³¹], thymectomy [¹³²], lymphadenectomy [¹³³], and intestinal anastomosis [¹³⁴]. Another application is to use NIRF as a positioning marker. For example, supervised autonomous suturing was performed on soft tissues under the guidance of NIRF markers [¹³⁵].

Spectral imaging acquires multiple images of the tissue at different wavelengths to reveal tissue characteristics. Multispectral imaging and hyperspectral imaging differ from each other in spectral resolution, band quantity, band width, and band contiguousness [¹³⁶]. Multispectral images are taken in a time span of several hundred milliseconds, during which the tissue and the camera shall remain steady with respect to each other, making it challenging to apply in a surgical setting [¹³⁷]. Deblurring algorithms were proposed to improve multispectral images affected by tissue/camera movements [¹³⁸]. Hyperspectral imaging was used to characterize oxygenation during robotic partial nephrectomy [¹³⁹]. Given that spectral imaging better reveals the tissue-specific optical characteristics compared with RGB imaging, it is also employed in automated tissue classification [¹²⁶,¹⁴⁰], as shown in Fig. 10B2.

Confocal endomicroscopy

Confocal endomicroscopy, which is an imaging technique for increasing resolution via the use of a spatial pinhole to reduce out-of-focus lights, enables in vivo histopathology by providing cellular level information and serves as an “optical biopsy.” The advantages of confocal endomicroscopy are evident: it provides high-resolution, real-time, dynamic images of tissues in a noninvasive manner and can be used to examine large areas during surgery. Probe-based confocal laser endomicroscopy (pCLE) [¹⁴¹] (Fig. 10C) is also promising due to its high-speed imaging, mosaicing algorithms, and robotic instrument/probe control [¹⁴²]. In particular, robot-assisted mosaicing (i.e., stitching adjacent image frames) has been widely investigated for large area tissue scanning using pCLE [¹²⁷,¹⁴³], whereas force feedback [¹⁴⁴] and visual servoing [¹⁴⁵] are used to ensure a stable and robust tissue contact. Projecting of the pCLE images back to the endoscopic footage was made possible using the tactile-based 3D surface reconstruction [¹⁴⁶] and SLAM [¹⁴⁷], allowing potential lesions to be conveniently targeted for subsequent treatments.

Control approaches

Most existing robotic systems for keyhole and endoscopic procedures utilize a master–slave teleoperation paradigm [¹⁴⁸], where the patient-side manipulators are on the slave side and the surgeon console is on the master side. The movements of the patient-side slave manipulators shall follow the trajectory commands from the master (usually haptic) devices. The advances in teleoperation are reported in the “Teleoperation” section. However, teleoperation has zero level of autonomy in a robot-assisted surgery [¹⁴⁹,¹⁵⁰]. Exploitations of autonomy on higher levels can be of greater assistance to surgeons, and the state-of-the-art progresses are summarized in the “System autonomy and surgical automation” section.

Teleoperation

Although the early studies on teleoperation for laparoscopy were conducted over long physical distances [¹⁵¹], in practice, the master and slave sides usually are in the same room. Currently, teleoperation can enhance surgeons’ capabilities, such as hand tremor elimination, motion scaling, and continued inputs for fully exploiting superhuman tool dexterity [³]. A review on master haptic devices is presented in the “Haptic devices” section, whereas the controls involved in teleoperation are discussed in the “Bilateral control” and “Shared control” sections.

Haptic devices

Haptic feedback includes cutaneous (tactile) and kinesthetic (force) information [¹⁵²], which can potentially improve the performance of delicate surgical tasks that involve interactions between the surgical manipulator and the environment (e.g., suturing or dissection) [⁹¹,¹⁵³]. Haptic feedback to operating surgeons is supported by haptic devices, which generate kinesthetic interactions for the surgeons to perceive the remote environment. Haptic devices sense the positions and/or orientations and generate forces and/or torques.

According to the capabilities of the sensing inputs and the haptic (force/torque) outputs, haptic devices can be divided into three categories as follows:

(1) 3-DoF inputs (usually positions) and 3-DoF outputs (usually forces): Commercial products include the delta.3 and omega.3 devices (Force Dimension) and the Novint Falcon (Novint Technologies). Research prototypes include the SHaDe device [¹⁵⁴] (orientation inputs and torque outputs), joystick mechanism [¹⁵⁵], and DELTA-R device [¹⁵⁶] (Fig. 11A).

(2) 6-DoF inputs (positions and orientations) and 3-DoF outputs (usually forces): Commercial products include the Touch and TouchX devices (3D Systems) and the omega.6 device (Force Dimension). Laparoscopic interface [¹⁵⁷] is one typical research prototype (Fig. 11B).

(3) 6-DoF inputs and multi-DoF outputs (five or six force/torque components): Commercial products include the delta.7 and sigma.7 devices (Force Dimension) and the Phantom Premium (3D Systems). Research prototypes include the haptic pen [¹⁵⁸], VISHARD6 [¹⁵⁹], PATHOS-II [¹⁶⁰], haptic cobot [¹⁶¹], modified DELTA-R device [¹⁵⁶], pinch–grasp haptic interface [¹⁶²], VirtuaPower device [¹⁶³] (Fig. 11C), and CombX device [¹⁶⁴].

Bilateral control

Two types of control architectures are commonly adopted in surgical teleoperation: unilateral and bilateral control. In unilateral control, the slave motions are directly specified by the master. Unilateral control is an effective and straightforward approach that has been widely implemented in robotic surgery systems (e.g., in the da Vinci system).

The main drawback of unilateral control is the absence of force feedback. Bilateral control, on the other hand, refers to the control of master–slave systems with force/position information exchange. The design considerations for the bilateral controller are to maintain the stability and transparency of the closed-loop system [¹⁶⁵]. Studies have shown that perfect transparency is not possible in practice as it requires the exact knowledge of the dynamics of master–slave system that transmits force and position in both directions without communication delay [¹⁶⁶]. The ideally transparent system is marginally stable, and increased stability robustness is achieved by reducing the bandwidth of accurate transparency, causing a trade-off between stability and transparency in the design of bilateral controllers [¹⁶⁷]. Due to the complexity of sensor integration and control analysis, control architectures with reduced number of channels for sensory information exchange are commonly adopted. In telesurgery, the slave is usually under position control, which leads to an impedance type or a direct force feedback type implementation of the bilateral controller.

The impedance control (also referred to as the position-error-based control [¹⁶⁸]) is a sensor-less control architecture that reflects a force from the difference between the desired and actual positions of the slave robot to the operator. This force is an indication of the interaction between the slave robot and the environment only when the friction and inertia are low [¹⁵²]. Therefore, this method is difficult to be applied in detecting interactions with soft tissue, and friction compensation is needed to improve the transparency of such bilateral systems [¹⁶⁹,¹⁷⁰].

Direct force feedback, on the other hand, requires the measurement of the interaction forces between the slave and the environment [¹⁷¹–¹⁷³]. It provides better performance in position and force tracking than position-error-based control [¹⁷⁴,¹⁷⁵]. However, practical challenges are encountered in force sensor integration due to miniaturization, sterilization, and biocompatibility issues.

Shared control

In shared control, the slave is teleoperated by the operator under an active assistance of the robot. A typical application of shared control in telesurgery is the use of virtual fixture [¹⁷⁶], which generates force and position signals as assistances to improve safety and accuracy, while the surgeon remains in control. Two types of virtual fixtures are commonly adopted: guidance virtual fixtures and forbidden-region virtual fixtures [¹⁷⁷]. For example, virtual fixtures have been used to provide guidance to target anatomy [¹⁷⁸] and motion constraints on robot-assisted suturing [¹⁷⁹]. Virtual fixtures have also been used for surgical training [¹⁸⁰] and co-manipulation [¹⁸¹]. User studies show that virtual fixtures improve surgical performances during procedures such as suturing, needle passing, and knot tying [¹⁸²,¹⁸³].

System autonomy and surgical automation

The current development of artificial intelligence is far from being capable to support autonomous surgery. Partial surgical autonomy can benefit surgical treatments by automating repetitive tasks and let surgeons concentrate on critical operations [¹⁵⁰].

The control architectures of teleoperated surgical systems can be classified as direct/bilateral control, shared control, and supervisory control depending on the degree of user interaction. Yang et al. proposed a framework of autonomy levels, in which medical robots have no autonomy, assistance autonomy, task autonomy, conditional autonomy, high autonomy, and full autonomy, corresponding to autonomy level 0 to 5, respectively [¹⁴⁹]. A similar six-level scaling is proposed in Reference [¹⁸⁴], which was developed from the discussions and a technical report of an ISO/IEC Joint Working Group.

Although supervised autonomy has been successfully applied in some robot-assisted surgeries such as orthopedic surgery and neurosurgery, the level of autonomy for keyhole surgery remains low (mostly teleoperation). With the growing number of surgical procedures with a massive quantity of available surgical data, standardized autonomy rating and classification are desired in order to guide the development of future surgical robots [¹⁴⁹,¹⁸⁵].

Surgical robotic systems with a high level of autonomy, as delineated by the above standards, are able to make (some) decisions under the supervision of a surgeon. The automation attempts are hindered by challenges in acquiring information and executing tasks in surgical environments with soft tissues or moving organs. Emerging studies have presented technical progresses in these aspects that potentially enable the development of next-generation cognitive surgical systems [¹⁸⁶]. In the rest of this section, the state-of-the-art surgical autonomy for keyhole surgeries will be reported.

Information acquisition

Visual modality provides a direct approach to the intracorporeal scene, which generally includes organs, tissues, and surgical instruments. Information acquisition refers to the extraction of informative segments from the scene and their interpretation as semantic or kinematic data, based on which the robotic surgical system executes surgical tasks autonomously.

Detecting the presences, determining the positions, and tracking the trajectories of surgical instruments are of importance not only in extending surgeons’ capabilities in teleoperation, but also in better facilitating autonomous surgery [¹⁸⁷]. The presence detection approaches adopted include color distinction [¹⁸⁸], geometric feature matching [¹⁸⁹], color and texture features [¹⁹⁰], and radiofrequency identification [¹⁹¹]. Positioning and tracking methods can be marker-based, for example, combining laser pointers and optical markers [¹⁹²] and using specially designed black/white patterns [¹⁹³]. Markerless approaches are also possible, for example, relying on the end effector geometry and online kinematic data for pose estimation [¹⁹⁴,¹⁹⁵]. Learning-based methods include a probabilistic condensation algorithm relying on the priori geometric knowledge of the instruments [¹⁹⁶], kinematics-combined randomized trees [¹⁹⁷,¹⁹⁸], shape-dependent feature descriptor for pose estimation [¹⁹⁹], and deep-learning-based instrument segmentation [²⁰⁰,²⁰¹].

Surgeries involving soft tissues have not been performed autonomously due to the lack of vision in tracking and distinguishing tissues in dynamic surgical environments [²⁰²]. Given that fluorescence imaging or multispectral imaging increases system complexity and cost, autonomous segmentation or tracking of organs using general endoscopic imaging is still actively investigated. In methods that utilize preoperative priors [²⁰³,²⁰⁴], the organs are usually segmented in the preoperative CT model, and the registration of the model to endoscopic images is performed to account for organs’ motion and deformation. Other works used image processing approaches or machine learning strategies without prior information. The image processing approaches include basic thresholding and merging [²⁰⁵], homogeneity and hue [²⁰⁶], gradient-based methods [²⁰⁷], and optical flow [²⁰⁸]. Segmentation based on machine learning has been achieved using random forest [²⁰⁹], support vector machines [²¹⁰], and fully convolutional neural networks [²¹¹]. This method is promising due to the increasing amount of available laparoscopic data.

Another topic is the analysis of instrument motions recorded from surgical procedures, which is also known as surgical task segmentation. It is extensively used for surgical skill assessment [²¹²,²¹³] or constructing finite state machines for automation [²¹⁴,²¹⁵]. Early studies have focused on supervised task segmentation [²¹⁶–²¹⁸], where a set of predefined surgical motion sets with explicit semantic sense is required. Given that laparoscopic surgical gestures do not follow predefined patterns and involve temporal variations, the supervised approaches lack the acuity in detecting surgeon’s gestures. In addition, manual annotation of the training data set is impractical when the data become massive. Hence, recent studies often treat the surgical task segmentation in an unsupervised fashion [²¹⁹–²²¹]. Improved robustness to looping (a.k.a. failures and repetitions in surgical procedures) and noises was reported based on evaluations on specific surgical tasks [²²¹] using unsupervised segmentation.

Autonomous planning and execution of surgical subtasks

Suturing and knot tying are two fundamental tasks in minimally invasive surgery (MIS) due to their time consumption and high risks of injuring the related unstructured in vivo environment. The autonomy focusing on these two tasks is indeed challenging considering the thread’s flexibility, position, and tension; tissue deformations; and constrained workspace.

The first robotic suturing in MIS was investigated in Reference [²²²]. EndoBot’s proposed autonomous robotic suturing algorithms were based on observations of manual suturing operations, which were divided into stitching, creating a suture loop, developing a knot, and securing a knot. To minimize the task uncertainty from the tissue deformations and the pose of the suture needle, other path planning approaches were subsequently proposed, including relying on kinematic analysis and geometric modeling of the stitching task [²²³], creating an analytical solution from manual suturing [²²⁴], and sequential convex programming [²²⁵].

To accommodate the uncertainty and adopt online change, a few alternative approaches were proposed by using (1) human guidance, where a laser pointer was maneuvered by a surgeon to pinpoint the entry for an automatic stitching under visual servoing [²²⁶]; (2) fluorescent imaging, where near-infrared fluorescent imaging was used to detect and track soft tissue deformations and automatically compute stitch arrangement [²⁰²,²²⁷]; and (3) tissue deformation modeling, where a penetration-induced deformation matrix was introduced to adaptively estimate suturing trajectories [²²⁸].

On the other hand, learning-based approaches were proposed for motion planning of surgical subtasks [¹⁵⁰]. Examples include the RNN-based autonomous knot tying using the EndoPAR robot [²²⁹], learning by demonstrations and decomposing demonstrations into meaningful primitives [²³⁰], apprenticeship learning under a variant of iterative learning control [²³¹], and non-rigid registration mapping between the demonstration scene and the test scene [²³²].

Besides suturing and knot tying, other autonomous surgical subtasks were also attempted, including multilateral debridement [²³³], multilateral cutting of 3D viscoelastic and 2D orthotropic tissue phantoms [²¹⁴], electro-surgery [²³⁴], palpation [²³⁵], and blunt dissection [²³⁶]. All these subtasks can be potentially fitted within the framework for hierarchical subtask execution planning, which was proposed in Reference [²³⁷]. Such an integrated effort is expected to eventually bring all these pilot autonomous functions into one working system.

Challenges and future perspectives

In multi-port surgical procedures, robotic assistance, which had brought high-definition stereo imaging, enhanced dexterity, and intuitive and fine control of instruments, gradually gained acceptance across the world with its applications in various surgical departments, including general surgery, urology, cardiothoracic surgery, and gynecology. Suitable technologies for imaging, system design, and robotic manipulation in multi-port procedures have also progressively arrived at consensus.

On the other hand, a movable vision module with illumination and two to three surgical manipulators have to be deployed to a surgical site via a straight or curved access channel in single-port and NOTES/endoscopic procedures, as shown in Fig. 3C. The clinical need for robotic assistance in these procedures is even more apparent than that for multi-port procedures. However, even though few systems have received clinical clearance (e.g., the da Vinci SP system), the effectiveness and durability of these robotic systems have yet to be tested. One fundamental reason is that the proximal joints in these intracorporeal manipulators are under extreme size constrain with a very high actuation requirement because an external load on the end effector has a much longer moment arm than that of the actuation force from the actuation element inside the manipulator. Furthermore, enough space and perhaps a central lumen for actuation of the distal joints and electrosurgical/mechanical end effectors (e.g., needle drivers and bipolar graspers) should be reserved in the proximal joints. It is hence extremely challenging to design and realize such an intracorporeal manipulator. Many technical approaches are still being actively explored as discussed in the “Systems for single-port procedures” and “Systems for endoscopic and NOTES procedures” sections.

A recent opinion is to design specific systems for otorhinolaryngologic, transurethral, or gastrointestinal procedures. Given the functional and manipulation similarity (i.e., one vision unit and two tele-operated manipulators), one general system can be more welcomed. However, the core problem is that whether such a universal, modular, and scalable solution (including materials, structures, and actuations) can be developed to fulfill all these needs.

Endoscopic imaging heavily depends on the technological readiness of chip fabrication techniques in industrial electronics. On the other hand, multispectral imaging, which only needs limited modification to existing endoscopic imaging system with added controls over exposure and illumination, would likely continue to increase its clinical uses, providing high-definition images on multiple scales to facilitate surgical operations.

Besides expensive and complex surgical robotic systems, intelligent hand-held surgical instruments (e.g., the ones in References [²³⁸,²³⁹]) can also trigger future studies. Light weight and control intuitiveness are of paramount importance for these instruments to gain clinical acceptance.

While exploring new designs to address the aforementioned needs in single-port and NOTES/endoscopic procedures, enough attention should be directed toward physical forms and control realizations of the new designs.

Physical forms mainly involve the utilized materials and adopted structure topologies. The intended reduction in instrument invasiveness with payload and precision requirements as high as possible needs a design that is compact and strong. This determines the need for materials with high Young’s modulus. Soft materials with low Young’s modulus may only fit in implantable devices that interact with organs and tissues on a macroscale (e.g., the soft heart sleeve [²⁴⁰]).

Articulated manipulators in single-port and endoscopic procedures may suffer from actuation deficiency. In cable-driven designs, proper tension keeping for adequate actuation can be challenging, whereas the use of embedded motors and rigid-linked linkages leads to design bulkiness and limited dexterity. Continuum mechanisms, on the other hand, deform the structure to transmit motions and forces. The dual roles of structural and transmission components can bring design compactness and expand the applications of continuum medical robots [¹¹]. Superelastic materials that allow large deformation shall be used. However, bending in a continuum structure has finite radius, which may not be able to deliver the required dexterity in tightly confined spaces. Using continuum-articulated hybrid structure, which utilizes the advantages from both structural topologies, may be a promising direction.

Regarding the control realization of future developments of surgical robotic systems, teleoperation will remain the main approach for the near future. Given the advances in communication technology (e.g., 5G), increased control information exchanges will be implemented, leading to increased teleoperation transparency.

Current artificial intelligence framework might only facilitate assistive surgical tasks, such as supervised suturing and knot tying. This may be due to the fact that the current data-driven approaches essentially generate an output by encoding thousands and voluminous particular samples. A new architect for reasoning is eventually needed to safely and properly handle patient-specific surgical operations.

Conclusions

In keyhole and endoscopic surgical procedures, robotics has greatly improved ergonomics, manipulation dexterity, and intuitiveness. The gradual global acceptance indicates the improved clinical outcomes. Given the history on how robotics has revolutionized various industrial sectors, the consensus is quite evident: the presence of robotics in medicine will inevitably increase. Future robotic assistance in surgery is expected to become a surgeon’s extended hands, eyes, and even mind in delivering delicate patient-specific treatments such that patients can benefit from the implementations of surgical robotic systems.

References

[1] Cuschieri A. Laparoscopic surgery: current status, issues and future developments. Surgeon 2005; 3(3): 125–130, 132–133, 135–138

[2] Alimoglu O, Sagiroglu J, Atak I, Kilic A, Eren T, Caliskan M, Bas G. Robot-assisted laparoscopic (RAL) procedures in general surgery. Int J Med Robot 2016; 12(3): 427–430

[3] Taylor RH. A perspective on medical robotics. Proc IEEE 2006; 94(9): 1652–1664

[4] Kuo CH, Dai JS, Dasgupta P. Kinematic design considerations for minimally invasive surgical robots: an overview. Int J Med Robot 2012; 8(2): 127–145

[5] Bergeles C, Yang GZ. From passive tool holders to microsurgeons: safer, smaller, smarter surgical robots. IEEE Trans Biomed Eng 2014; 61(5): 1565–1576

[6] Smith JA, Jivraj J, Wong R, Yang V. 30 years of neurosurgical robots: review and trends for manipulators and associated navigational systems. Ann Biomed Eng 2016; 44(4): 836–846

[7] Jacofsky DJ, Allen M. Robotics in arthroplasty: a comprehensive review. J Arthroplasty 2016; 31(10): 2353–2363

[8] Kim C, Ryu SC, Dupont PE. Real-time adaptive kinematic model estimation of concentric tube robots. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2015: 3214–3219

[9] Bergeles C, Gosline AH, Vasilyev NV, Codd PJ, Del Nido PJ, Dupont PE. Concentric tube robot design and optimization based on task and anatomical constraints. IEEE Trans Robot 2015; 31(1): 67–84

[10] Elhawary H, Tse ZTH, Hamed A, Rea M, Davies BL, Lamperth MU. The case for MR-compatible robotics: a review of the state of the art. Int J Med Robot 2008; 4(2): 105–113

[11] Burgner-Kahrs J, Rucker DC, Choset H. Continuum robots for medical applications: a survey. IEEE Trans Robot 2015; 31(6): 1261–1280

[12] Beasley RA. Medical robots: current systems and research directions. J Robot 2012; 2012: 1–14

[13] Diana M, Marescaux J. Robotic surgery. Br J Surg 2015; 102(2): e15–e28

[14] Rassweiler JJ, Autorino R, Klein J, Mottrie A, Goezen AS, Stolzenburg JU, Rha KH, Schurr M, Kaouk J, Patel V, Dasgupta P, Liatsikos E. Future of robotic surgery in urology. BJU Int 2017; 120(6): 822–841

[15] Navarra G, Pozza E, Occhionorelli S, Carcoforo P, Donini I. One-wound laparoscopic cholecystectomy. Br J Surg 1997; 84(5): 695

[16] Kalloo AN, Singh VK, Jagannath SB, Niiyama H, Hill SL, Vaughn CA, Magee CA, Kantsevoy SV. Flexible transgastric peritoneoscopy: a novel approach to diagnostic and therapeutic interventions in the peritoneal cavity. Gastrointest Endosc 2004; 60(1): 114–117

[17] Guthart GS, Salisbury JK. The Intuitive^TM Telesurgery System: overview and application. IEEE International Conference on Robotics and Automation (ICRA) 2000: 618–621

[18] Abdel Raheem A, Troya IS, Kim DK, Kim SH, Won PD, Joon PS, Hyun GS, Rha KH. Robot-assisted Fallopian tube transection and anastomosis using the new REVO-I robotic surgical system: feasibility in a chronic porcine model. BJU Int 2016; 118(4): 604–609

[19] Taylor R, Jensen P, Whitcomb L, Barnes A, Kumar R, Stoianovici D, Gupta P, Wang Z, Dejuan E, Kavoussi L. A steady-hand robotic system for microsurgical augmentation. Int J Robot Res 1999; 18(12): 1201–1210

[20] Li J, Zhou N, Wang S, Gao Y, Liu D. Design of an integrated master-slave robotic system for minimally invasive surgery. Int J Med Robot 2012; 8(1): 77–84

[21] Freschi C, Ferrari V, Melfi F, Ferrari M, Mosca F, Cuschieri A. Technical review of the da Vinci surgical telemanipulator. Int J Med Robot 2013; 9(4): 396–406

[22] Kim U, Lee DH, Kim YB, Seok DY, So J, Choi HR. S-Surge: novel portable surgical robot with multiaxis force-sensing capability for minimally invasive surgery. IEEE/ASME Trans Mechatron 2017; 22(4): 1717–1727

[23] Lum MJH, Friedman DCW, Sankaranarayanan G, King H, Fodero K, Leuschke R, Hannaford B, Rosen J, Sinanan MN. The RAVEN: design and validation of a telesurgery system. Int J Robot Res 2009; 28(9): 1183–1197

[24] Hannaford B, Rosen J, Friedman DW, King H, Roan P, Cheng L, Glozman D, Ma J, Kosari SN, White L. Raven-II: an open platform for surgical robotics research. IEEE Trans Biomed Eng 2013; 60(4): 954–959

[25] Kuo CH, Dai JS. Kinematics of a fully-decoupled remote center-of-motion parallel manipulator for minimally invasive surgery. J Med Device 2012; 6(2): 021008

[26] Mitsuishi M, Sugita N, Pitakwatchara P. Force feedback augmentation modes in the laparoscopic minimal invasive telesurgical system. IEEE/ASME Trans Mechatron 2007; 12(4): 447–454

[27] Berkelman P, Ma J. A compact modular teleoperated robotic system for laparoscopic surgery. Int J Robot Res 2009; 28(9): 1198–1215

[28] Dombre E, Michelin M, Pierrot F, Poignet P, Bidaud P, Morel G, Ortmaier T, Salle D, Zemiti N, Gravez P, Karouia M, Bonnet N. MARGE project: design, modelling, and control of assistive devices for minimally invasive surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2004; LNCS 3217: 1–8

[29] Hagn U, Nickl M, Jörg S, Passig G, Bahls T, Nothhelfer A, Hacker F, Le-Tien L, Albu-Schäffer A, Konietschke R, Grebenstein M, Warpup R, Haslinger R, Frommberger M, Hirzinger G. The DLR MIRO: a versatile lightweight robot for surgical applications. Industrial Robot 2008; 35(4): 324–336

[30] Lopez E, Kwok KW, Payne CJ, Giataganas P, Yang GZ. Implicit active constraints for robot-assisted arthroscopy. IEEE International Conference on Robotics and Automation (ICRA) 2013: 5390–5395

[31] Dai Z, Wu Z, Zhao J, Xu K. A robotic laparoscopic tool with enhanced capabilities and modular actuation. Sci China Technol Sci 2019; 62(1): 47–59

[32] Çavuşoğlu MC, Cohn M, Tendick F, Sastry S. A laparoscopic telesurgical workstation. IEEE Trans Robot Autom 1999; 15(4): 728–739

[33] Sekimoto M, Nishikawa A, Taniguchi K, Takiguchi S, Miyazaki F, Doki Y, Mori M. Development of a compact laparoscope manipulator (P-arm). Surg Endosc 2009; 23(11): 2596–2604

[34] Dalvand MM, Shirinzadeh B. Remote centre-of-motion control algorithms of 6-RRCRR parallel robot assisted surgery system (PRAMiSS). IEEE International Conference on Robotics and Automation (ICRA) 2012: 3401–3406

[35] Çavuşoğlu MC, Williams W, Tendick F, Sastry SS. Robotics for telesurgery: second generation Berkeley/UCSF laparoscopic telesurgical workstation and looking towards the future applications. Industrial Robot 2003; 30(1): 22–29

[36] Nasseri MA, Gschirr P, Eder M, Nair S, Kobuch K, Maier M, Zapp D, Lohmann C, Knoll A. Virtual fixture control of a hybrid parallel-serial robot for assisting ophthalmic surgery: an experimental study. IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BIOROB) 2014: 732–738

[37] Azimian H, Patel RV, Naish MD. On constrained manipulation in robotics-assisted minimally invasive surgery. IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BIOROB) 2010: 650–655

[38] Sandoval J, Poisson G, Vieyres P. A new kinematic formulation of the RCM constraint for redundant torque-controlled robots. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2017: 4576–4581

[39] Funda J, Taylor RH, Eldridge B, Gomory S, Gruben KG. Constrained cartesian motion control for teleoperated surgical robots. IEEE Trans Robot Autom 1996; 12(3): 453–465

[40] Locke RCO, Patel RV. Optimal remote center-of-motion location for robotics-assisted minimally-invasive surgery. IEEE International Conference on Robotics and Automation (ICRA) 2007: 1900–1905

[41] Aghakhani N, Geravand M, Shahriari N, Vendittelli M, Oriolo G. Task control with remote center of motion constraint for minimally invasive robotic surgery. IEEE International Conference on Robotics and Automation (ICRA) 2013: 5807–5812

[42] Michelin M, Poignet P, Dombre E. Dynamic task/posture decoupling for minimally invasive surgery motions: simulation results. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2004: 3625–3630

[43] Cha HJ, Yi BJ. Modeling of a constraint force at RCM point in a needle insertion task. IEEE International Conference on Mechatronics and Automation (ICMA) 2011: 2177–2182

[44] Taylor RH, Funda J, Eldridge B, Gomory S, Gruben K, LaRose D, Talamini M, Kavoussi L, Anderson J. A telerobotics assistant for laparoscopic surgery. IEEE Eng Med Biol Mag 1995; 14(3): 279–288

[45] Bunter SE, Ghodoussi M. Transforming a surgical robot for human telesurgery. IEEE Trans Robot Autom 2003; 19(5): 818–824

[46] Wang W, Li J, Wang S, Su H, Jiang X. System design and animal experiment study of a novel minimally invasive surgical robot. Int J Med Robot 2016; 12(1): 73–84

[47] Ikuta K, Daifu S, Hasegawa T, Higashikawa H. Hyper-finger for remote minimally invasive surgery in deep area. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2002; 2488: 173–181

[48] Dachs GW, Peine WJ. A novel surgical robot design: minimizing the operating envelope within the sterile field. Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS) 2006: 1505–1508

[49] Nakamura Y, Matsui A, Saito T, Yoshimoto K. Shape-memory-alloy active forceps for laparoscopic surgery. IEEE International Conference on Robotics and Automation (ICRA) 1995: 2320–2327

[50] Shi ZY, Liu D, Wang TM. A shape memory alloy-actuated surgical instrument with compact volume. Int J Med Robot 2014; 10(4): 474–481

[51] Takahashi H, Warisawa S, Mitsuishi M, Arata J, Hashizume M. Development of high dexterity minimally invasive surgical system with augmented force feedback capability. IEEE / RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BIOROB) 2006: 284–289

[52] Hong MB, Jo YH. Design of a novel 4-DOF wrist-type surgical instrument with enhanced rigidity and dexterity. IEEE/ASME Trans Mechatron 2014; 19(2): 500–511

[53] Robinson G, Davies JB. Continuum robots—a state of the art. IEEE International Conference on Robotics and Automation (ICRA) 1999: 2849–2853

[54] Francis P, Eastwood KW, Bodani V, Price K, Upadhyaya K, Podolsky D, Azimian H, Looi T, Drake J. Miniaturized instruments for the da Vinci research kit: design and implementation of custom continuum tools. IEEE Robot Autom Mag 2017; 24(2): 24–33

[55] Abbott DJ, Becke C, Rothstein RI, Peine WJ. Design of an endoluminal NOTES robotic system. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2007: 410–416

[56] Kaouk JH, Haber GP, Autorino R, Crouzet S, Ouzzane A, Flamand V, Villers A. A novel robotic system for single-port urologic surgery: first clinical investigation. Eur Urol 2014; 66(6): 1033–1043

[57] Xu K, Zhao J, Fu M. Development of the SJTU unfoldable robotic system (SURS) for single port laparoscopy. IEEE/ASME Trans Mechatron 2015; 20(5): 2133–2145

[58] Arata J, Fujisawa Y, Nakadate R, Kiguchi K, Harada K, Mitsuishi M, Hashizume M. Compliant four degree-of-freedom manipulator with locally deformable elastic elements for minimally invasive surgery. IEEE International Conference on Robotics and Automation (ICRA) 2019: 2663–2669

[59] Tobushi H, Furuichi Y, Sakuragi T, Sugimoto Y. Bending fatigue properties of a superelastic thin tube and a high-elastic thin wire of TiNi alloy. Mater Trans 2009; 50(8): 2043–2049

[60] Vaida C, Plitea N, Pisla D, Gherman B. Orientation module for surgical instruments—a systematical approach. Meccanica 2013; 48(1): 145–158

[61] Kanno T, Haraguchi D, Yamamoto M, Tadano K, Kawashima K. A forceps manipulator with flexible 4-DOF mechanism for laparoscopic surgery. IEEE/ASME Trans Mechatron 2015; 20(3): 1170–1178

[62] Sa Z, Chen Y, Li Q, Zhao B, Xu K. Kinematic optimization of a continuum surgical manipulator. IEEE International Conference on Robotics and Biomimetics (ROBIO) 2018: 2069–2074

[63] Wu Z, Li Q, Zhao J, Gao J, Xu K. Design of a modular continuum-articulated laparoscopic robotic tool with decoupled kinematics. IEEE Robot Autom Lett 2019; 4(4): 3545–3552

[64] Morelli L, Guadagni S, Di Franco G, Palmeri M, Di Candio G, Mosca F. Da Vinci single site© surgical platform in clinical practice: a systematic review. Int J Med Robot 2016; 12(4): 724–734

[65] Choi H, Kwak HS, Lim YA, Kim HJ. Surgical robot for single-incision laparoscopic surgery. IEEE Trans Biomed Eng 2014; 61(9): 2458–2466

[66] Seeliger B, Diana M, Ruurda JP, Konstantinidis KM, Marescaux J, Swanström LL. Enabling single-site laparoscopy: the SPORT platform. Surg Endosc 2019; 33(11): 3696–3703

[67] Walker AS, Steele SR. The future of robotic instruments in colon and rectal surgery. Semin Colon Rectal Surg 2016; 27(3): 144–149

[68] Roh KS, Yoon S, Do Kwon Y, Shim Y, Kim YJ. Single-port surgical robot system with flexible surgical instruments. International Conference on Intelligent Robotics and Applications (ICIRA) 2015: 447–459

[69] Petroni G, Niccolini M, Menciassi A, Dario P, Cuschieri A. A novel intracorporeal assembling robotic system for single-port laparoscopic surgery. Surg Endosc 2013; 27(2): 665–670

[70] Wortman TD, Mondry JM, Farritor SM, Oleynikov D. Single-site colectomy with miniature in vivo robotic platform. IEEE Trans Biomed Eng 2013; 60(4): 926–929

[71] Yung KL, Cheung JLK, Chung SW, Singh S, Yeung CK. A single-port robotic platform for laparoscopic surgery with a large central channel for additional instrument. Ann Biomed Eng 2017; 45(9): 2211–2221

[72] Kobayashi Y, Sekiguchi Y, Noguchi T, Takahashi Y, Liu Q, Oguri S, Toyoda K, Uemura M, Ieiri S, Tomikawa M, Ohdaira T, Hashizume M, Fujie MG. Development of a robotic system with six-degrees-of-freedom robotic tool manipulators for single-port surgery. Int J Med Robot 2015; 11(2): 235–246

[73] Cheon B, Gezgin E, Ji DK, Tomikawa M, Hashizume M, Kim HJ, Hong J. A single port laparoscopic surgery robot with high force transmission and a large workspace. Surg Endosc 2014; 28(9): 2719–2729

[74] Xu K, Goldman RE, Ding J, Allen PK, Fowler DL, Simaan N. System design of an insertable robotic effector platform for single port access (SPA) surgery. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2009: 5546–5552

[75] Wang H, Zhang R, Chen W, Wang X, Pfeifer R. A cable-driven soft robot surgical system for cardiothoracic endoscopic surgery: preclinical tests in animals. Surg Endosc 2017; 31(8): 3152–3158

[76] Zhao J, Feng B, Zheng MH, Xu K. Surgical robots for SPL and NOTES: a review. Minim Invasive Ther Allied Technol 2015; 24(1): 8–17

[77] Mintz Y, Horgan S, Cullen J, Stuart D, Falor E, Talamini MA. NOTES: a review of the technical problems encountered and their solutions. J Laparoendosc Adv Surg Tech A 2008; 18(4): 583–587

[78] Ikuta K, Yamamoto K, Sasaki K. Development of remote microsurgery robot and new surgical procedure for deep and narrow space. IEEE International Conference on Robotics and Automation (ICRA) 2003: 1103–1108

[79] Phee SJ, Low SC, Sun ZL, Ho KY, Huang WM, Thant ZM. Robotic system for no-scar gastrointestinal surgery. Int J Med Robot 2008; 4(1): 15–22

[80] Lehman AC, Dumpert J, Wood NA, Redden L, Visty AQ, Farritor S, Varnell B, Oleynikov D. Natural orifice cholecystectomy using a miniature robot. Surg Endosc 2009; 23(2): 260–266

[81] Tortora G, Dario P, Menciassi A. Array of robots augmenting the kinematics of endocavitary surgery. IEEE/ASME Trans Mechatron 2014; 19(6): 1821–1829

[82] Son J, Cho CN, Kim KG, Chang TY, Jung H, Kim SC, Kim MT, Yang N, Kim TY, Sohn DK. A novel semi-automatic snake robot for natural orifice transluminal endoscopic surgery: preclinical tests in animal and human cadaver models (with video). Surg Endosc 2015; 29(6): 1643–1647

[83] Clark J, Noonan DP, Vitiello V, Sodergren MH, Shang J, Payne CJ, Cundy TP, Yang GZ, Darzi A. A novel flexible hyper-redundant surgical robot: prototype evaluation using a single incision flexible access pelvic application as a clinical exemplar. Surg Endosc 2015; 29(3): 658–667

[84] Zhao J, Zheng X, Zheng M, Shih AJ, Xu K. An endoscopic continuum testbed for finalizing system characteristics of a surgical robot for NOTES procedures. IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM) 2013: 63–70

[85] Goldman RE, Bajo A, MacLachlan LS, Pickens R, Herrell SD, Simaan N. Design and performance evaluation of a minimally invasive telerobotic platform for transurethral surveillance and intervention. IEEE Trans Biomed Eng 2013; 60(4): 918–925

[86] Burgner J, Rucker DC, Gilbert HB, Swaney PJ, Russell PT 3rd, Weaver KD, Webster RJ 3rd. A telerobotic system for transnasal surgery. IEEE/ASME Trans Mechatron 2014; 19(3): 996–1006

[87] Thakkar S, Awad M, Gurram KC, Tully S, Wright C, Sanan S, Choset H. A novel, new robotic platform for natural orifice distal pancreatectomy. Surg Innov 2015; 22(3): 274–282

[88] Mitchell CR, Hendrick RJ, Webster RJ 3rd, Herrell SD. Toward improving transurethral prostate surgery: development and initial experiments with a prototype concentric tube robotic platform. J Endourol 2016; 30(6): 692–696

[89] Légner A, Diana M, Halvax P, Liu YY, Zorn L, Zanne P, Nageotte F, De Mathelin M, Dallemagne B, Marescaux J. Endoluminal surgical triangulation 2.0: a new flexible surgical robot. Preliminary pre-clinical results with colonic submucosal dissection. Int J Med Robot 2017; 13(3): e1819

[90] Xu K, Liang B, Dai Z, Zhao J, Zhao B, Liu H, Xiao L, Sun Y. Preliminary development of a continuum dual-arm surgical robotic system for transurethral procedures. International Conference on Intelligent Robotics and Applications (ICIRA) 2017: 311–322

[91] Wagner CR, Howe RD. Force feedback benefit depends on experience in multiple degree of freedom robotic surgery task. IEEE Trans Robot Autom 2007; 23(6): 1235–1240

[92] Wagner CR, Stylopoulos N, Howe RD. The role of force feedback in surgery: analysis of blunt dissection. Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS) 2002: 68–74

[93] Xu K, Simaan N. An investigation of the intrinsic force sensing capabilities of continuum robots. IEEE Trans Robot 2008; 24(3): 576–587

[94] Hong MB, Jo YH. Design and evaluation of 2-DOF compliant forceps with force-sensing capability for minimally invasive robot surgery. IEEE Trans Robot 2012; 28(4): 932–941

[95] Sokhanvar S, Packirisamy M, Dargahi J. A multifunctional PVDF-based tactile sensor for minimally invasive surgery. Smart Mater Struct 2007; 16(4): 989–998

[96] Zarrin PS, Escoto A, Xu R, Patel RV, Naish MD, Trejos AL. Development of an optical fiber-based sensor for grasping and axial force sensing. IEEE International Conference on Robotics and Automation (ICRA) 2017: 939–944

[97] Kim U, Kim YB, So J, Seok D, Choi HR. Sensorized surgical forceps for robotic-assisted minimally invasive surgery. IEEE Trans Ind Electron 2018; 65(12): 9604–9613

[98] He C, Wang S, Sang H, Li J, Zhang L. Force sensing of multiple-DOF cable-driven instruments for minimally invasive robotic surgery. Int J Med Robot 2014; 10(3): 314–324

[99] Lee D, Kim U, Gulrez T, Yoon WJ, Hannaford B, Choi HR. A laparoscopic grasping tool with force sensing capability. IEEE/ASME Trans Mechatron 2016; 21(1): 130–141

[100] Tholey G, Desai JP, Castellanos AE. Force feedback plays a significant role in minimally invasive surgery results and analysis. Ann Surg 2005; 241(1): 102–109

[101] Seibold U, Kubler B, Hirzinger G. Prototype of instrument for minimally invasive surgery with 6-axis force sensing capability. IEEE International Conference on Robotics and Automation (ICRA) 2005: 496–501

[102] Moradi Dalvand M, Shirinzadeh B, Shamdani AH, Smith J, Zhong Y. An actuated force feedback-enabled laparoscopic instrument for robotic-assisted surgery. Int J Med Robot 2014; 10(1): 11–21

[103] Peirs J, Clijnen J, Reynaerts D, Brussel HV, Herijgers P, Corteville B, Boone S. A micro optical force sensor for force feedback during minimally invasive robotic surgery. Sens Actuators A Phys 2004; 115(2): 447–455

[104] Noh Y, Sareh S, Würdemann H, Liu H, Back J, Housden J, Rhode K, Althoefer K. Three-axis fiber-optic body force sensor for flexible manipulators. IEEE Sens J 2016; 16(6): 1641–1651

[105] Haslinger R, Leyendecker P, Seibold U. A fiberoptic force-torque-sensor for minimally invasive robotic surgery. 2013 IEEE International Conference on Robotics and Automation 2013: 4390–4395

[106] Qasaimeh MA, Sokhanvar S, Dargahi J, Kahrizi M. PVDF-based microfabricated tactile sensor for minimally invasive surgery. J Microelectromech Syst 2009; 18(1): 195–207

[107] Xu K, Simaan N. Intrinsic wrench estimation and its performance index of multi-segment continuum robots. IEEE Trans Robot 2010; 26(3): 555–561

[108] Lee SJ, Lee SC, Ahn HS. Design and control of tele-matched surgery robot. Mechatronics 2014; 24(5): 395–406

[109] Kuebler B, Seibold U, Hirzinger G. Development of actuated and sensor integrated forceps for minimally invasive robotic surger. Int J Med Robot 2005; 1(3): 96–107

[110] Baumhauer M, Feuerstein M, Meinzer HP, Rassweiler J. Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol 2008; 22(4): 751–766

[111] Schols RM, Bouvy ND, van Dam RM, Stassen LP. Advanced intraoperative imaging methods for laparoscopic anatomy navigation: an overview. Surg Endosc 2013; 27(6): 1851–1859

[112] Stoyanov D, Mylonas GP, Deligianni F, Darzi A, Yang GZ. Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2005: 139–146

[113] Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD. Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 2009; 73(4): 896–900

[114] Röhl S, Bodenstedt S, Suwelack S, Dillmann R, Speidel S, Kenngott H, Muller-Stich BP. Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. Med Phys 2012; 39(3): 1632–1645

[115] Clancy NT, Stoyanov D, Maier-Hein L, Groch A, Yang GZ, Elson DS. Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging. Biomed Opt Express 2011; 2(11): 3119–3128

[116] Schmalz C, Forster F, Schick A, Angelopoulou E. An endoscopic 3D scanner based on structured light. Med Image Anal 2012; 16(5): 1063–1072

[117] Le HND, Nguyen H, Wang Z, Opfermann J, Leonard S, Krieger A, Kang JU. Demonstration of a laparoscopic structured-illumination three-dimensional imaging system for guiding reconstructive bowel anastomosis. J Biomed Opt 2018; 23(5): 1–10

[118] Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D. Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 2013; 17(8): 974–996

[119] Bartoli A, Gerard Y, Chadebecq F, Collins T. On template-based reconstruction from a single view: analytical solutions and proofs of wellposedness for developable, isometric and conformal surfaces. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2012: 2026–2033

[120] Wu C, Narasimhan SG, Jaramaz B. A multi-image shape-from-shading framework for near-lighting perspective endoscopes. Int J Comput Vis 2010; 86(2–3): 211–228

[121] Mountney P, Stoyanov D, Davison A, Yang GZ. Simultaneous stereoscope localization and soft-tissue mapping for minimal invasive surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2006: 347–354

[122] Mountney P, Yang GZ. Motion compensated SLAM for image guided surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2010: 496–504

[123] Mahmoud N, Cirauqui I, Hostettler A, Doignon C, Soler L, Marescaux J, Montiel JMM. ORBSLAM-based endoscope tracking and 3D reconstruction. International Workshop on Computer-Assisted and Robotic Endoscopy (CARE) 2016: 72–83

[124] Song J, Wang J, Zhao L, Huang S, Dissanayake G. MIS-SLAM: real-time large-scale dense deformable SLAM system in minimal invasive surgery based on heterogeneous computing. IEEE Robot Autom Lett 2018; 3(4): 4068–4075

[125] Daskalaki D, Aguilera F, Patton K, Giulianotti PC. Fluorescence in robotic surgery. J Surg Oncol 2015; 112(3): 250–256

[126] Regeling B, Thies B, Gerstner AOH, Westermann S, Müller NA, Bendix J, Laffers W. Hyperspectral imaging using flexible endoscopy for laryngeal cancer detection. Sensors (Basel) 2016; 16(8): E1288

[127] Erden MS, Rosa B, Boularot N, Gayet B, Morel G, Szewczyk J. Conic-spiraleur: a miniature distal scanner for confocal microlaparoscope. IEEE/ASME Trans Mechatron 2014; 19(6): 1786–1798

[128] Frangioni JV. In vivo near-infrared fluorescence imaging. Curr Opin Chem Biol 2003; 7(5): 626–634

[129] Tobis S, Knopf J, Silvers C, Yao J, Rashid H, Wu G, Golijanin D. Near infrared fluorescence imaging with robotic assisted laparoscopic partial nephrectomy: initial clinical experience for renal cortical tumors. J Urol 2011; 186(1): 47–52

[130] Bjurlin MA, Gan M, McClintock TR, Volpe A, Borofsky MS, Mottrie A, Stifelman MD. Near-infrared fluorescence imaging: emerging applications in robotic upper urinary tract surgery. Eur Urol 2014; 65(4): 793–801

[131] Spinoglio G, Priora F, Bianchi PP, Lucido FS, Licciardello A, Maglione V, Grosso F, Quarati R, Ravazzoni F, Lenti LM. Real-time near-infrared (NIR) fluorescent cholangiography in single-site robotic cholecystectomy (SSRC): a single-institutional prospective study. Surg Endosc 2013; 27(6): 2156–2162

[132] Wagner OJ, Louie BE, Vallières E, Aye RW, Farivar AS. Near-infrared fluorescence imaging can help identify the contralateral phrenic nerve during robotic thymectomy. Ann Thorac Surg 2012; 94(2): 622–625

[133] Holloway RW, Bravo RAM, Rakowski JA, James JA, Jeppson CN, Ingersoll SB, Ahmad S. Detection of sentinel lymph nodes in patients with endometrial cancer undergoing robotic-assisted staging: a comparison of colorimetric and fluorescence imaging. Gynecol Oncol 2012; 126(1): 25–29

[134] Jafari MD, Lee KH, Halabi WJ, Mills SD, Carmichael JC, Stamos MJ, Pigazzi A. The use of indocyanine green fluorescence to assess anastomotic perfusion during robotic assisted laparoscopic rectal surgery. Surg Endosc 2013; 27(8): 3003–3008

[135] Decker RS, Shademan A, Opfermann JD, Leonard S, Kim PCW, Krieger A. Biocompatible near-infrared three-dimensional tracking system. IEEE Trans Biomed Eng 2017; 64(3): 549–556

[136] Li Q, He X, Wang Y, Liu H, Xu D, Guo F. Review of spectral imaging technology in biomedical engineering: achievements and challenges. J Biomed Opt 2013; 18(10): 100901

[137] Stoyanov D. Surgical vision. Ann Biomed Eng 2012; 40(2): 332–345

[138] Lian J, Zheng Y, Jiao W, Yan F, Zhao B. Deblurring sequential ocular images from multi-spectral imaging (MSI) via mutual information. Med Biol Eng Comput 2018; 56(6): 1107–1113

[139] Olweny EO, Faddegon S, Best SL, Jackson N, Wehner EF, Tan YK, Zuzak KJ, Cadeddu JA. Renal oxygenation during robot-assisted laparoscopic partial nephrectomy: characterization using laparoscopic digital light processing hyperspectral imaging. J Endourol 2013; 27(3): 265–269

[140] Moccia S, Wirkert SJ, Kenngott H, Vemuri AS, Apitz M, Mayer B, De Momi E, Mattos LS, Maier-Hein L. Uncertainty-aware organ classification for surgical data science applications in laparoscopy. IEEE Trans Biomed Eng 2018; 65(11): 2649–2659

[141] Gmitro AF, Aziz D. Confocal microscopy through a fiber-optic imaging bundle. Opt Lett 1993; 18(8): 565–567

[142] Zuo S, Yang GZ. Endomicroscopy for computer and robot assisted intervention. IEEE Rev Biomed Eng 2017; 10: 12–25

[143] Zuo S, Hughes M, Seneci C, Chang TP, Yang GZ. Toward intraoperative breast endomicroscopy with a novel surface-scanning device. IEEE Trans Biomed Eng 2015; 62(12): 2941–2952

[144] Giataganas P, Hughes M, Yang GZ. Force adaptive robotically assisted endomicroscopy for intraoperative tumour identification. Int J CARS 2015; 10(6): 825–832

[145] Rosa B, Erden MS, Vercauteren T, Herman B, Szewczyk J, Morel G. Building large mosaics of confocal edomicroscopic images using visual servoing. IEEE Trans Biomed Eng 2013; 60(4): 1041–1049

[146] Giataganas P, Vitiello V, Simaiaki V, Lopez E, Yang GZ. Cooperative in situ microscopic scanning and simultaneous tissue surface reconstruction using a compliant robotic manipulator. IEEE International Conference on Robotics and Automation (ICRA) 2013: 5378–5383

[147] Mountney P, Giannarou S, Elson D, Yang GZ. Optical biopsy mapping for minimally invasive cancer screening. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2009: 483–490

[148] Avgousti S, Christoforou EG, Panayides AS, Voskarides S, Novales C, Nouaille L, Pattichis CS, Vieyres P. Medical telerobotic systems: current status and future trends. Biomed Eng Online 2016; 15(1): 96

[149] Yang GZ, Cambias J, Cleary K, Daimler E, Drake J, Dupont PE, Hata N, Kazanzides P, Martel S, Patel RV, Santos VJ, Taylor RH. Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci Robot 2017; 2(4): eaam8638

[150] Yip M, Das N. Robot autonomy for surgery. The Encyclopedia of Medical Robotics. World Scientific Publishing, 2018: 281–313

[151] Rovetta A, Sala R, Cosmi F, Wen X, Sabbadini D, Milanesi S, Togno A, Angelini L, Bejczy A. The first experiment in the world of robotic telesurgery for laparoscopy carried out by means of satellites networks and optical fibres networks on 7th July 1993. International Conference on Industrial Electronics, Control, and Instrumentation (IECON) 1993; 1: 51–56

[152] Okamura AM. Methods for haptic feedback in teleoperated robot-assisted surgery. Ind Rob 2004; 31(6): 499–508

[153] Talasaz A, Trejos AL, Patel RV. Effect of force feedback on performance of robotics-assisted suturing. IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob) 2012: 823–828

[154] Birglen L, Gosselin C, Pouliot N, Monsarrat B, Laliberté T. SHaDe, a new 3-DOF haptic device. IEEE Trans Robot Autom 2002; 18(2): 166–175

[155] Steger R, Lin K, Adelstein BD, Kazerooni H. Design of a passively balanced spatial linkage haptic interface. J Mech Des 2004; 126(6): 984–991

[156] Arata J, Kondo H, Ikedo N, Fujimoto H. Haptic device using a newly developed redundant parallel mechanism. IEEE Trans Robot 2011; 27(2): 201–214

[157] Tholey G, Desai JPA. General-purpose 7 DOF haptic device: applications toward robot-assisted surgery. IEEE/ASME Trans Mechatron 2007; 12(6): 662–669

[158] Stocco LJ, Salcudean SE, Sassani F. Optimal kinematic design of a haptic pen. IEEE/ASME Trans Mechatron 2001; 6(3): 210–220

[159] Ueberle M, Buss M. Design, control, and evaluation of a new 6 DOF haptic device. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2002: 2949–2954

[160] Kim K, Kyun W. Design and analysis of a new 7-DOF parallel type haptic device: PATHOS-II. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2003: 2241–2246

[161] Faulring EL, Colgate JE, Peshkin MA. The cobotic hand controller: design, control and performance of a novel haptic display. Int J Robot Res 2006; 25(11): 1099–1119

[162] Najdovski Z, Nahavandi S, Fukuda T. Design, development, and evaluation of a pinch–grasp haptic interface. IEEE/ASME Trans Mechatron 2014; 19(1): 45–54

[163] Lee G, Hur SM, Oh Y. A novel haptic device with high-force display capability and wide workspace. IEEE International Conference on Robotics and Automation (ICRA) 2016: 2704–2709

[164] Zhao B, Sa Z, Wu Z, Li Q, Xu K. CombX: design of a haptic device for teleoperation. IEEE International Conference on Robotics and Biomimetics (ROBIO) 2018: 1969–1974

[165] Hokayem PF, Spong MW. Bilateral teleoperation: an historical survey. Automatica 2006; 42(12): 2035–2057

[166] Hashtrudi-Zaad K, Salcudean SE. Analysis of control architectures for teleoperation systems with impedance/admittance master and slave manipulators. Int J Robot Res 2001; 20(6): 419–445

[167] Lawrence DA. Stability and transparency in bilateral teleoperation. IEEE Trans Robot Autom 1993; 9(5): 624–637

[168] Tavakoli M, Aziminejad A, Patel RV, Moallem M. High-fidelity bilateral teleoperation systems and the effect of multimodal haptics. IEEE Trans Syst Man Cybern B Cybern 2007; 37(6): 1512–1528

[169] Mahvash M, Okamura A. Friction compensation for enhancing transparency of a teleoperator with compliant transmission. IEEE Trans Robot 2007; 23(6): 1240–1246

[170] Torabi A, Khadem M, Zareinia K, Sutherland GR, Tavakoli M. Application of a redundant haptic interface in enhancing soft-tissue stiffness discrimination. IEEE Robot Autom Lett 2019; 4(2): 1037–1044

[171] Preusche C, Ortmaier T, Hirzinger G. Teleoperation concepts in minimal invasive surgery. Control Eng Pract 2002; 10(11): 1245–1250

[172] Tavakoli M, Patel RV, Moallem M. Haptic interaction in robot-assisted endoscopic surgery: a sensorized end-effector. Int J Med Robot 2005; 1(2): 53–63

[173] Kim U, Seok DY, Kim YB, Lee DH, Choi HR. Development of a grasping force-feedback user interface for surgical robot system. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2016: 845–850

[174] Cavusoglu MC, Sherman A, Tendick F. Design of bilateral teleoperation controllers for haptic exploration and telemanipulation of soft environments. IEEE Trans Robot Autom 2002; 18(4): 641–647

[175] Tavakoli M, Howe RD. Haptic effects of surgical teleoperator flexibility. Int J Robot Res 2009; 28(10): 1289–1302

[176] Rosenberg LB. Virtual fixtures: perceptual tools for telerobotic manipulation. Proceedings of IEEE Virtual Reality Annual International Symposium 1993: 76–82

[177] Abbott JJ, Marayong P, Okamura AM. Haptic virtual fixtures for robot-assisted manipulation. Robotics Research 2007; 28: 49–64

[178] Lopez E, Zollo L, Guglielmelli E. Teleoperated control based on virtual fixtures for a redundant surgical system. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013: 450–455

[179] Kapoor A, Li M, Taylor RH. Spatial motion constraints for robot assisted suturing using virtual fixtures. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2005: 89–96

[180] Shahbazi M, Atashzar SF, Patel RV. A dual-user teleoperated system with virtual fixtures for robotic surgical training. IEEE International Conference on Robotics and Automation (ICRA) 2013: 3639–3644

[181] Vitrani MA, Poquet C, Morel G. Applying virtual fixtures to the distal end of a minimally invasive surgery instrument. IEEE Trans Robot 2017; 33(1): 114–123

[182] Chen Z, Malpani A, Chalasani P, Deguet A, Vedula SS, Kazanzides P, Taylor RH. Virtual fixture assistance for needle passing and knot tying. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2016: 2343–2350

[183] Fontanelli GA, Yang GZ, Siciliano B. A comparison of assistive methods for suturing in MIRS. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2018: 4389–4395

[184] Haidegger T. Autonomy for surgical robots: concepts and paradigms. IEEE Transactions on Medical Robotics and Bionics 2019; 1(2): 65–76

[185] Moustris GP, Hiridis SC, Deliparaschos KM, Konstantinidis KM. Evolution of autonomous and semi-autonomous robotic surgical systems: a review of the literature. Int J Med Robot 2011; 7(4): 375–392

[186] Hoeckelmann M, Rudas IJ, Fiorini P, Kirchne F, Haidegger T. Current capabilities and development potential in surgical robotics. Int J Adv Robot Syst 2015; 12(5): 61

[187] Münzer B, Schoeffmann K, Böszörmenyi L. Content-based processing and analysis of endoscopic images and videos: a survey. Multimedia Tools Appl 2018; 77(1): 1323–1362

[188] Wei GQ, Arbter K, Hirzinger G. Real-time visual servoing for laparoscopic surgery. Controlling robot motion with color image segmentation. IEEE Eng Med Biol Mag 1997; 16(1): 40–45

[189] Voros S, Long JA, Cinquin P. Automatic detection of instruments in laparoscopic images: a first step towards high-level command of robotic endoscopic holders. Int J Robot Res 2007; 26(11–12): 1173–1190

[190] Pezzementi Z, Voros S, Hager GD. Articulated object tracking by rendering consistent appearance parts. IEEE International Conference on Robotics and Automation (ICRA) 2009: 3940–3947

[191] Kranzfelder M, Schneider A, Fiolka A, Schwan E, Gillen S, Wilhelm D, Schirren R, Reiser S, Jensen B, Feussner H. Real-time instrument detection in minimally invasive surgery using radiofrequency identification technology. J Surg Res 2013; 185(2): 704–710

[192] Krupa A, Gangloff J, Doignon C, de Mathelin MF, Morel G, Leroy J, Soler L, Marescaux J. Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Trans Robot Autom 2003; 19(5): 842–853

[193] Nageotte F, Zanne P, Doignon C, de Mathelin M. Visual servoing-based endoscopic path following for robot-assisted laparoscopic surgery. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2006: 2364–2369

[194] Reiter A, Allen PK, Zhao T. Feature classification for tracking articulated surgical tools. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2012: 592–600

[195] Ye M, Zhang L, Giannarou S, Yang GZ. Real-time 3D tracking of articulated tools for robotic surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2016: 386–394

[196] Wolf R, Duchateau J, Cinquin P, Voros S. 3D tracking of laparoscopic instruments using statistical and geometric modeling. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2011: 203–210

[197] Allan M, Ourselin S, Thompson S, Hawkes DJ, Kelly J, Stoyanov D. Toward detection and localization of instruments in minimally invasive surgery. IEEE Trans Biomed Eng 2013; 60(4): 1050–1058

[198] Reiter A, Allen PK, Zhao T. Appearance learning for 3D tracking of robotic surgical tools. Int J Robot Res 2014; 33(2): 342–356

[199] Reiter A, Goldman RE, Bajo A, Iliopoulos K, Simaan N, Allen PK. A learning algorithm for visual pose estimation of continuum robots. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2011: 2390–2396

[200] García-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Poorten EV, Stoyanov D, Vercauteren T, Ourselin SE. Toolnet: holistically-nested real-time segmentation of robotic surgical tools. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2017: 5717–5722

[201] Shvets AA, Rakhlin A, Kalinin AA, Iglovikov VI. Automatic instrument segmentation in robot-assisted surgery using deep learning. IEEE International Conference on Machine Learning and Applications (ICMLA) 2018: 624–628

[202] Shademan A, Decker RS, Opfermann JD, Leonard S, Krieger A, Kim PCW. Supervised autonomous robotic soft tissue surgery. Sci Transl Med 2016; 8(337): 337ra64

[203] Nakamoto M, Ukimura O, Gill IS, Mahadevan A, Miki T, Hashizume M, Sato Y. Realtime organ tracking for endoscopic augmented reality visualization using miniature wireless magnetic tracker. International Workshop on Medical Imaging and Virtual Reality 2008: 359–366

[204] Nosrati MS, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R, Hamarneh G. Efficient multi-organ segmentation in multi-view endoscopic videos using pre-operative priors. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2014: 324–331

[205] Bilodeau GA, Shu Y, Cheriet F. Multistage graph-based segmentation of thoracoscopic images. Comput Med Imaging Graph 2006; 30(8): 437–446

[206] Tjoa MP, Krishnan SM, Kugean C, Wang P, Doraiswaini R. Segmentation of clinical endoscopic image based on homogeneity and hue. International Conference of the IEEE Engineering in Medicine and Biology Society 2001: 2665–2668

[207] Figueiredo IN, Moreno JC, Prasath VBS, Figueiredo PN. A segmentation model and application to endoscopic images. International Conference Image Analysis and Recognition 2012: 164–171

[208] Wu S, Nakao M, Matsuda T. Continuous lung region segmentation from endoscopic images for intra-operative navigation. Comput Biol Med 2017; 87(1): 200–210

[209] Bodenstedt S, Wagner M, Mayer B, Stemmer K, Kenngott H, Müller-Stich B, Dillmann R, Speidel S. Image-based laparoscopic bowel measurement. Int J CARS 2016; 11(3): 407–419

[210] Chhatkuli A, Bartoli A, Malti A, Collins T. Live image parsing in uterine laparoscopy. International Symposium on Biomedical Imaging 2014: 1263–1266

[211] Moccia S, Foti S, Rossi SM, Rota I, Scotti M, Toffoli S, Mattos LS, Momi ED, Frontoni E. FCNN-based segmentation of kidney vessels—towards constraints definition for safe robot-assisted nephrectomy. Joint Workshop on New Technologies for Computer/Robot Assisted Surgery 2018

[212] Rosen J, Hannaford B, Richards CG, Sinanan MN. Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 2001; 48(5): 579–591

[213] Lin HC, Shafran I, Yuh D, Hager GD. Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comput Aided Surg 2006; 11(5): 220–230

[214] Murali A, Sen S, Kehoe B, Garg A, McFarland S, Patil S, Boyd WD, Lim S, Abbeel P, Goldberg K. Learning by observation for surgical subtasks: multilateral cutting of 3D viscoelastic and 2D orthotropic tissue phantoms. IEEE International Conference on Robotics and Automation (ICRA) 2015: 1202–1209

[215] Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N. Statistical modeling and recognition of surgical workflow. Med Image Anal 2012; 16(3): 632–641

[216] Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, Haro BB, Zappella L, Khudanpur S, Vidal R, Hager GD. A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng 2017; 64(9): 2025–2041

[217] Tao L, Elhamifar E, Khudanpur S, Hager GD, Vidal R. Sparse hidden Markov models for surgical gesture classification and skill evaluation. International Conference on Information Processing in Computer-Assisted Interventions 2012: 167–177

[218] Zappella L, Béjar B, Hager G, Vidal R. Surgical gesture classification from video and kinematic data. Med Image Anal 2013; 17(7): 732–745

[219] Despinoy F, Bouget D, Forestier G, Penet C, Zemiti N, Poignet P, Jannin P. Unsupervised trajectory segmentation for surgical gesture recognition in robotic training. IEEE Trans Biomed Eng 2016; 63(6): 1280–1291

[220] Murali A, Garg A, Krishnan S, Pokorny FT, Abbeel P, Darrell T, Goldberg K. TSC-DL: unsupervised trajectory segmentation of multi-modal surgical demonstrations with deep learning. IEEE International Conference on Robotics and Automation (ICRA) 2016: 4150–4157

[221] Krishnan S, Garg A, Patil S, Lea C, Hager G, Abbeel P, Goldberg K. Transition state clustering: unsupervised surgical trajectory segmentation for robot learning. Int J Robot Res 2017; 36(13–14): 1595–1618

[222] Kang H, Wen JT. Endobot: a robotic assistant in minimally invasive surgeries. IEEE Int Conf Robot Autom 2001; 2: 2031–2036 (ICRA)

[223] Nageotte F, Zanne P, Doignon C, de Mathelin M. Stitching planning in laparoscopic surgery: towards robot-assisted suturing. Int J Robot Res 2009; 28(10): 1303–1321

[224] Jackson RC, Çavuşoğlu MC. Needle path planning for autonomous robotic surgical suturing. IEEE International Conference on Robotics and Automation (ICRA) 2013: 1669–1675

[225] Sen S, Garg A, Gealy DV, McKinley S, Jen Y, Goldberg K. Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization. IEEE International Conference on Robotics and Automation (ICRA) 2016: 4178–4185

[226] Staub C, Osa T, Knoll A, Bauernschmitt R. Automation of tissue piercing using circular needles and vision guidance for computer aided laparoscopic surgery. IEEE International Conference on Robotics and Automation (ICRA) 2010: 4585–4590

[227] Leonard S, Wu KL, Kim Y, Krieger A, Kim PCW. Smart tissue anastomosis robot (STAR): a vision-guided robotics system for laparoscopic suturing. IEEE Trans Biomed Eng 2014; 61(4): 1305–1317

[228] Zhong F, Wang Y, Wang Z, Liu YH. Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing. IEEE Robot Autom Lett 2019; 4(3): 2669–2676

[229] Mayer H, Gomez F, Wierstra D, Nagy I, Knoll A, Schmidhuber J. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2006: 543–548

[230] Mayer H, Nagy I, Burschka D, Knoll A, Braun EU, Lange R, Bauernschmitt R. Automation of manual tasks for minimally invasive surgery. International Conference on Autonomic and Autonomous Systems (ICAS) 2008: 260–265

[231] Jvd B. Miller S, Duckworth D, Hu H, Wan A, Fu X-Y, Goldberg K, Abbeel P. Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations. IEEE International Conference on Robotics and Automation (ICRA) 2010: 2074–2081

[232] Schulman J, Gupta A, Venkatesan S, Tayson-Frederick M, Abbeel P. A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2013: 4111–4117

[233] Kehoe B, Kahn G, Mahler J, Kim J, Lee A, Lee A, Nakagawa K, Patil S, Boyd WD, Abbeel P, Goldberg K. Autonomous multilateral debridement with the raven surgical robot. IEEE International Conference on Robotics and Automation (ICRA) 2014: 1432–1439

[234] Le HND, Opfermann JD, Kam M, Raghunathan S, Saeidi H, Leonard S, Kang JU, Krieger A. Semi-autonomous laparoscopic robotic electro-surgery with a novel 3D Endoscope. IEEE International Conference on Robotics and Automation (ICRA) 2018: 6637–6644

[235] Nichols KA, Okamura AM. Autonomous robotic palpation: machine learning techniques to identify hard inclusions in soft tissues. IEEE International Conference on Robotics and Automation (ICRA) 2013: 4384–4389

[236] Nagy DÁ, Nagy TD, Elek R, Rudas IJ, Haidegger T. Ontology-based surgical subtask automation, automating blunt dissection. J Med Robot Res 2018; 3(03n04): 1841005

[237] Nagy TD, Haidegger T. A DVRK-based framework for surgical subtask automation. Acta Polytech Hung 2019; 16(8): 61–78

[238] Awtar S, Trutna TT, Nielsen JM, Abani R, Geiger J. FlexDex™: a minimally invasive surgical tool with enhanced dexterity and intuitive control. J Med Device 2010; 4(3): 035003–035011

[239] Miyazaki R, Hirose K, Ishikawa Y, Kanno T, Kawashima K. A master-slave integrated surgical robot with active motion transformation using wrist axis. IEEE/ASME Trans Mechatron 2018; 23(3): 1215–1225

[240] Roche ET, Horvath MA, Wamala I, Alazmani A, Song S-E, Whyte W, Machaidze Z, Payne CJ, Weaver JC, Fishbein G, Kuebler J, Vasilyev NV, Mooney DJ, Pigula FA, Walsh CJ. Soft robotic sleeve supports heart function. Sci Transl Med 2017; 9(373): eaaf3925

RIGHTS & PERMISSIONS

Higher Education Press